Data Execution Prevention (DEP) is a feature used by some operating systems, including Microsoft Windows® that is used to reduce the risks of exploits that rely on, for example, storing code to stack or heap memory locations that normally do not contain executable code. DEP typically works by marking some regions of a computer readable memory as “non-executable”. This ensures that those regions can only store data that are recognized as being non-executable. A successful exploitation is therefore rendered less likely to happen as it may try to run code from a region of the memory that is marked as non-executable. The operating system detects this and prevents the code from being executed.
In order to attempt to bypass DEP, some forms of exploit use Return-Oriented Programming (ROP). In one example, the exploit relies on overwriting a return address in the memory stack that has called by a valid application so that it points to code other than code associated with the normal application execution path. This code can be used for malicious purposes. An additional portion of the stack is overwritten that allows the exploit to call pre-existing functions without needing to inject malicious code into the program. This type of attack only uses existing executable code. ROP exploits can be used to chain individual small attacks. In other words, the stack memory is used indirectly to execute previously selected instructions (referred to as gadgets). Each gadget typically ends with the x86 subroutine return instruction (RET), which transfers the execution to the next gadget or to the payload. As all of the executed instructions are executed from executable memory areas within the original application, DEP is ineffective. The gadgets typically include code that is equivalent to calling a VirtualProtect API to enable execution rights for the memory page where the payload resides, which effectively renders DEP ineffective.
There are several mechanisms designed to protect against an ROP exploit, which are described in detail in P. Bania, “Security Mitigations for Return-Oriented Programming Attacks”.
A technique to address ROP exploits is Address Space Layout randomization (ASLR). ASLR randomly arranges locations in the memory at which data such as executable files and libraries are stored. This makes it more difficult for an ROP exploit to predict a target address. For example, an ROP exploit must locate the code to be executed, which requires knowledge of the memory address of the code. ASLR takes advantage of the low probability of an exploit knowing where regions of the memory are randomly located. For an exploit to be successful, it must know the locations of all required regions of the memory, which is extremely unlikely.
However, not all software applications have executables and libraries that can make use ASLR, and so these are more vulnerable to ROP exploitation. An example of an occurrence of this type is described in Common Vulnerabilities and Exposures number CVE-2010-2883, where the ROP exploit evaded the DEP.